Experimental Study and Modeling of Three Classes of Collective Problem-Solving Methods

People working together can be very successful problem-solvers. Many real-life examples, from Wikipedia to citizen science projects, show that, under the right conditions, crowds can find remarkable solutions to complex problems. Yet, joining the capabilities of many people can be challenging. What factors make some groups more successful than others? How does the nature of the problem and the structure of the environment influence the group's performance? To answer these questions, I consider problem-solving as a search process -- a situation in which individuals are searching for a good solution. I describe and compare three different methods for structuring groups: (1) non-interacting groups, where individuals search independently without exchanging any information, (2) social groups, where individuals freely exchange information during their search, and (3) solution-influenced groups, where individuals repeatedly contribute to a shared collective solution. First, I introduce the idea of transmission chains - a specific type of solution-influenced group where individuals tackle the problem one after another, each one starting from the solution of its predecessor. I apply this method to binary choice problems and compare it to majority voting rules in non-interacting groups. The results show that transmission chains are superior in environments where individual accuracy is low and confidence is a reliable indicator of performance. This type of environment, however, is rarely observed in two experimental datasets. Then, I evaluate the performance of transmission chains for problems that have a complex structure, such as multidimensional optimization tasks. Again, I use non-interacting groups as a comparison, this time by selecting the best out of multiple independent solutions. Simulations and experimental data show that transmission chains outperform independent groups under two environmental conditions: either when problems are rather easy, or when group members are relatively unskilled. Next, I focus on social groups, where individuals influence each other during the search. To understand the social dynamics that operate in such groups, I conduct two studies: I first examine how people search for a solution independently from others, and then study how this individual process is impacted by social influence. The first study presents experimental data to show that the individual search behavior can be described by a take-the-best heuristic, that is, a simple rule-of-thumb that ignores all but one cue at a time. This heuristic reproduces a variety of behavioral patterns observed in different environments. Then, I extend this heuristic to include social interactions where multiple individuals exchange information during their search. My results show that, in this case, individuals tend to converge towards similar solutions. This induces a collective search dilemma: compared to non-interacting groups, the quality of the average individual's solution is improved at the expense of the best solution of the group. Nevertheless, further analyses show that this dilemma disappears for more difficult problems. Overall, this thesis shows that no collective problem-solving method is superior to the others in all environments and for all problems. Instead, the performance of each method depends on numerous factors, such as the nature of the task, the problem difficulty, the group composition, and the skill levels of the individuals. My work helps understanding the role of these different factors and their influence on collective problem-solving

Can simple transmission chains foster collective intelligence in binary-choice tasks?

In many social systems, groups of individuals can find remarkably efficient solutions to complex cognitive problems, sometimes even outperforming a single expert. The success of the group, however, crucially depends on how the judgments of the group members are aggregated to produce the collective answer. A large variety of such aggregation methods have been described in the literature, such as averaging the independent judgments, relying on the majority or setting up a group discussion. In the present work, we introduce a novel approach for aggregating judgments - the transmission chain - which has not yet been consistently evaluated in the context of collective intelligence. In a transmission chain, all group members have access to a unique collective solution and can improve it sequentially. Over repeated improvements, the collective solution that emerges reflects the judgments of every group members. We address the question of whether such a transmission chain can foster collective intelligence for binary-choice problems. In a series of numerical simulations, we explore the impact of various factors on the performance of the transmission chain, such as the group size, the model parameters, and the structure of the population. The performance of this method is compared to those of the majority rule and the confidence-weighted majority. Finally, we rely on two existing datasets of individuals performing a series of binary decisions to evaluate the expected performances of the three methods empirically. We find that the parameter space where the transmission chain has the best performance rarely appears in real datasets. We conclude that the transmission chain is best suited for other types of problems, such as those that have cumulative properties

LIONESS Lab: a free web-based platform for conducting interactive experiments online

LIONESS Lab is a free web-based platform for interactive online experiments. An intuitive, user-friendly graphical interface enables researchers to develop, test, and share experiments online, with minimal need for programming experience. LIONESS Lab provides solutions for the methodological challenges of interactive online experimentation, including ways to reduce waiting time, form groups on-the-fly, and deal with participant dropout. We highlight key features of the software, and show how it meets the challenges of conducting interactive experiments online

Impact of the group size <i>N</i>.

<p>(A) The color-coding indicates the probability of success of the chain method as a function of the group size <i>N</i> and the contribution threshold <i>τ</i>. The column of values at <i>N</i> = 10 corresponds to the red curve shown in <b><a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0167223#pone.0167223.g003" target="_blank">Fig 3</a></b>. (B) Comparison of the performance of the chain method with the contribution threshold set to <i>τ</i> = 0.85 (in red), the majority rule (in grey), and the weighted-majority rule (in blue). These results are computed assuming a proportion of correct answer <i>q</i>₁ = 0.6 and the confidence distributions shown in <b><a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0167223#pone.0167223.g001" target="_blank">Fig 1</a></b>.</p

Description of the environment.

<p>Assumed distributions of confidence among the individuals who provide the correct answer to the problem (in blue), and among those who provide a wrong answer to the problem (in red). The interval of confidence values ranges from <i>c</i> = 0 (very uncertain) to <i>c</i> = 1 (very certain). The blue and red distributions are beta distributions with shape parameters <i>α</i>₁ = 8 and <i>β</i>₁ = 2 (mean value: 0.8), and <i>α</i>₀ = 3 and <i>β</i>₀ = 3 (mean value: 0.5), respectively. In the simulations, a proportion <i>q</i>₁ of the sample population gives the correct answer and have confidence levels drawn from the blue distribution, and a proportion <i>q</i>₀ = 1 − <i>q</i>₁ of the sample population gives a wrong answer and have confidence levels drawn from the red distribution. In empirical data, the shape parameters of the blue and red distributions depend on the nature of the task.</p

Population structure and corresponding best method for two binary-choice task studies.

<p>(A) Experimental participants evaluating which of two cities has a larger population. The task is repeated across 1000 different pairs of cities. Each point in the graph corresponds to one instance of the task (i.e., one pair of cities). (B) Dermoscopists evaluating 108 cases of skin lesions and evaluating whether the lesion is cancerous or not-cancerous. Each point in the graph corresponds to one medical case. In (A) and (B) the position of each point indicates the proportion of respondents (i.e., participants or doctors) who provided the correct answer (<i>y</i>-axis), and the confidence utility (<i>x</i>-axis) for that case. The border colour of each point indicates the aggregation method that performs best in this particular case (the majority in blue, the weighted-majority in red, and the chain in green). Cases for which several methods perform equally good are represented in blue if the majority rule is one of them and in red if the weighted-majority is one of them. In addition, the grey-scale colour inside each point indicates the success chance of the best performing method, ranging from 0 (in black) to 1 (in white). These results are calculated assuming 1000 groups of <i>N</i> = 10 individuals randomly sampled from the pool of available respondents.</p

Illustration of a transmission chain.

<p>For this case study, we assume a group size of <i>N</i> = 10, and a proportion of correct answers <i>q</i>₁ = 0.6. (A) The <i>N</i> individuals are randomly drawn from the sample population and assigned to a random position in the chain (the red and blue dots). Among them, five individuals have the correct answer (i.e., the blue dots at chain positions 1, 3, 5, 6, and 10), and five individuals have a wrong answer (i.e., the red dots at chain position 2, 4, 7, 8, and 9). The black dashed line represents the activity threshold <i>τ</i> = 0.6 indicating the confidence level above which individuals contribute to the collective solution. The “contributors” replace the current collective solution by their own solution. Individuals who do not contribute (i.e., those with a confidence level lower than <i>τ</i>) leave the collective solution unchanged. (B) The resulting collective solution in the chain at each position. The blue open circles indicate a correct answer, and the red open circles indicate a wrong one. In this example, the individual at chain position 1 initialises the collective solution with a correct answer. The collective solution remains unchanged until the contributor at chain position 4 replaces it by a wrong answer. The individual at position 6 is also a contributor and restores the correct answer. All other individuals have no impact on the collective solution. In this example, the chain generates a correct solution.</p

Impact of the contribution threshold <i>τ</i>.

<p>The red line indicates the probability that the chain generates a correct solution for different values of the contribution threshold <i>τ</i>, assuming a group size of <i>N</i> = 10, a proportion of correct answers <i>q</i>₁ = 0.6 and the confidence distributions shown in <b><a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0167223#pone.0167223.g001" target="_blank">Fig 1</a></b>. In these conditions, the optimal value for the contribution threshold is <i>τ</i> = 0.83, for which the chain produces the correct solution 93% of the time. The grey and blue lines indicate the success chances of the majority and the weighted-majority rules, respectively.</p